Creative Design in a Tangible User Interface Environment

This paper shows that tangible user interfaces support cognitive actions that are associated with creative design. The evidence for this is a case study of designers using a tangible user interface environment for manipulating 3D models on a digital design workbench. Focussing on how the new interface technology changes designers’ spatial cognition, we compare designers using tangible user interfaces with designers using graphical user interfaces in a collaborative design task. The results show that the combination of tangible interaction with Augmented Reality display techniques improve designers’ perception of spatial relationships between 3D models and encourages designers to discover hidden spatial features. These characteristics of designing are associated with creative design. 1. Spatial Cognition and Creativity in Design Creativity is generally characterised by aesthetic appeal, novelty, quality, unexpectedness, uncommonness, peer-recognition, influence, intelligence, learning, and popularity (Runco and Prizker 1998). Thus, creativity in the design process is associated with discoveries and ideas that are fundamentally novel, where designers discover hidden features in a representation and recognise a key concept as a sudden insight. We expect that a new tangible user interface environment for design can play a critical role in the creative design process by improving designers’ spatial cognition. The changes of designers’ perception of spatial knowledge when using tangible user interfaces might lead to such discoveries and to the production of creative ideas. We consider the existing digital workbenches as defining a class of design environments that use tangible user interfaces (TUIs) to be a departure from the traditional graphical user interfaces (GUIs) that designers are currently using to create and interact with digital design models. 2 M. J. KIM AND M. L. MAHER We associate a designer’s perception of the form and spatial relationships of the design components with the designer’s spatial cognition. In our research, the meaning of ‘space’ to the designers is not an abstract of empty space, but rather of the identity and the relative locations of the objects in space. Space then is decomposed into particular objects and the spatial relationships among them. The spatial relationships may include functional issues since designing attempts to satisfy intended functions. Thus, we investigate designers’ spatial cognition or improved understanding of the form and spatial relationships between 3D objects with a focus on unexpected discoveries. This paper presents the results of a case study using protocol analysis. 1.1. CREATIVE PROBLEM SOLVING IN DESIGN Gestalt theorists have emphasised productive thinking in contrast with reproductive thinking in the domain of creative problem solving (Wertheimer 1982). Productive thinking depends on past experience in only a general way and involves new structural understanding of the specific requirements of a problem. Gestalt analysis of creative thinking indicates the negative influences of past experiences on creative thinking. On the other hand, reproductive thinking theorists argue that the important issue for creative problem solving is not to abandon reproductive thinking itself but to reorganise the past experience for the current situation. Reproductive thinking applies some past knowledge to a present problem directly. Creative thinking is closely associated with the concepts of restructuring, which may form the basis for insight into the problem (Ohlsson 1984). Weisberg (Weisberg 1982) posed the results of case studies that creative thinking moves beyond established practises only slowly as a modification of the past rather than rejection of the past. Cross and Dorst proposed that creative design can be modelled in terms of the co-evolution of problem and solution spaces, as described by Maher et al. (Dorst and Cross 2001; Maher 1996). That is, creative design involves a period of exploration in which both the formulation of the problem and ideas for its solution are developed and refined together, with constant iteration of analysis, synthesis and evaluation processes between the two ‘spaces’. Accordingly, a creative event occurs as the moment of insight at which a problem-solution pair is framed in a potentially resolvable form, where the designer’s ability of framing a design problem is emphasised as a key aspect of creativity. They introduce the notion of ‘default’ and ‘surprise’ problem/solution space to describe creative design, which keeps a designer from routine behaviour by leading to framing and reframing of the design CREATIVE DESIGN IN A TANGIBLE USER INTERFACE ENVIRONMENT3 problem. The common characteristic of creative thinking in these studies is the restructuring of information available to the designer while designing. 1.2. COGNITIVE ACTIONS FOR CREATIVE DESIGN: INSIGHT AND UNEXPECTED DISCOVERIES Sometimes, people suddenly realise the answers during problem solving, even though they cannot figure out how to get to the solution (Davison 1995). The occurrence of “insight” associated with this ‘Aha!’ experience is one of characteristic features of creativity in design (Akin 1990). There are two conventional views of insight; the “special-process” views and the “nothing-special” views. Included in the special-process view is the idea that insight results from a restructuring of a problem that is accompanied by an unconscious leap in thinking, that it results from greatly accelerated mental processing, and that it is due to a short-circuiting of normal reasoning processes (Perkins 1981). In contrast, the nothing-special view proposes that insight is merely an extension of ordinary processes of perceiving, recognising, learning, and conceiving (Perkins 1981). Here we focus on the restructuring of a problem, a change in a person’s perception of a problem situation, where the contribution of ‘unexpected discoveries’ is stressed. According to Suwa et al., “unexpected discoveries” refer to designers’ perceptual actions of attending to implicit visuo-spatial features in sketches that are discovered in an unexpected way by later inspection (Suwa 2000). Designers sometimes notice consequences that were not intended when they drew (Schön and Wiggins 1992). They also argue that designers do not just synthesise solutions that satisfy initially given requirements but also invent design issues or requirements that capture important aspects of the given problem, and call this ‘situated-invention (S-invention)’. In terms of coevolution, unexpected discoveries can be regarded as the act of finding new aspects of the developing solution-space and S-invention can be regarded as the act of expanding the problem-space. They found that unexpected discoveries of visuo-spatial features in sketches and S-inventions become the strong impetus for the occurrences of each other by using protocol analysis. The findings provide empirical evidence for the co-evolution view. Research in design cognition has primarily dealt with 2D sketches, so we interpreted concepts and findings from studies of designing with 2D sketches in terms of 3D design and then applied them to our research. Since characteristics of creative design can be modelled in terms of the coevolution of problem and solution spaces, we look for designers’ restructuring a problem and their exploration of the problem space and the solution space. More specifically, we look for unexpected discoveries and Sinvention in designing using TUI and GUI environments. 4 M. J. KIM AND M. L. MAHER 2. Spatial Cognition While Using Tangible Interfaces to Digital Design Models TUIs are new approaches to human-computer interaction that are often associated with “augmented reality” (AR). Since AR technology blends reality and virtuality, TUIs combine physical and digital worlds, which allow very different “reflective conversation” between the two environments (Arias et al. 1997). Above all, TUIs provide a tangible interaction by turning the physical objects into input and output devices for computer interfaces. The strengths of physical interaction can be explained by two aspects; direct, naïve manipulability and tactile interaction as an additional dimension of interaction. Thus, they enable designers to create and interact with digital models that go beyond the traditional human-computer interface of the keyboard and mouse. The tangible interactions using TUIs in AR systems can be explained by the concept of “augmented affordance”, posed by Seichter and Kvan (Seichter and Kvan 2004). From this point of view, TUIs can be seen as offering a conduit between the real or perceived affordances implied by the physical properties of the interface tool and the affordances created by the digital behaviours in the virtualised interface. The term “affordance” refers to the perceived and actual properties of the thing that determine just how the thing could possibly be used, which results from the mental interpretation of things based on our past knowledge and experience applied to our perception of the things (Gibson 1979; Norman 1988). We predict that tangible interaction in TUIs account for changes in the designers’ spatial cognition of 3D digital models. 2.1. DESIGNERS’ SPATIAL COGNITION As a consequence of the diversity of approaches and related disciplines, there is little consistency in what is meant by the term “spatial” (Foreman and Gillett 1997). In this research we define a designer’s spatial cognition as the designer’s perception of the form and spatial relationships of the objects or spaces in 3D design. Associated with the physical interaction, touch is emphasised as a spatial modality linking motor and spatial processes closely while using TUIs to digital models. Kinaesthetic information through a haptic system provides us with the ability to construct a spatial map of objects that we touch (Loomis and Lederman 1986). It is the movement of a hand repeatedly colliding with objects that comes to define extra-personal space for each individual, as a consequence of repeatedly experienced associations (Foreman and Gillett 1997). Thus, the movement simulated by the mouse in desk-top systems lacks tactile and kinaesthetic feedback that normally accompanies movement. CREATIVE DESIGN IN A TANGIBLE USER INTERFACE ENVIRONMENT5 Language draws on spatial cognition so that designers can talk about what they perceive and it thereby provides a window on the nature of spatial cognition (Anibaldi and Nualláin 1998). It is based on the assumption that people often use general purpose verbs and prepositions when the context is sufficiently clear to disambiguate them. Thus, we analyse the designers’ conversation in order to investigate their spatial cognition. Gesture is also recognized as a good vehicle for capturing visual and spatial information as it is associated with visuo-spatial content. Furthermore, the movement of hands can facilitate recall of visuo-spatial items as well as verbal items (Wagner 2004). People produce some gestures along with their speech, and such speech-accompanying gestures are not just hand moving. Speech and gesture are both characterising the spatial relationships among entities, which are closely related to and may even be beneficial for cognitive processing (Goldin-Meadow 2003; Lavergne and Kimura 1987). 2.2. DIGITAL DESIGN WORKBENCHES We reviewed various digital design workbenches: metaDESK, iNavigator, BUILD-IT, PSyBench, URP, MIXdesign and ARTHUR system. The metaDESK system was constructed by Ulmer and Ishii (Ullmer and Ishii 1997) with a focus on physical interaction to manipulate the digital environment. Standard 2D GUI elements like icons, and menus, are given a physical instantiation as wooden frames, phicons, and trays, respectively. iNavigator is a CAD platform for designers to navigate and construct 3D models, which consists of a vertical tablet device for displaying a dynamic building section view and a horizontal table surface for displaying the corresponding building plan. The display tablet is served as “a cutting plane” (Lee et al. 2003). BUILD-IT developed by Fjeld et al. (Fjeld 1998) is a cooperative planning tool consisting of a table, bricks and a screen, which allows a group of designers, co-located around the table, to interact, by means of physical bricks, with models in a virtual 3D setting. A plan view of the scene is projected onto the table and a perspective view of the scene is projected on the wall. Brave et al. (Brave et al. 1999) designed PSyBench and inTouch, employing tele-manipulation technology to create the illusion of shared physical objects that distant users are interacting with. Although still in the early stage, it shows the potential of distributed tangible interfaces. URP developed by MIT media lab is a luminous tangible workbench for urban planning that integrates functions addressing a broad range of the field’s concerns such as cast shadows, reflections and wind-flow into a single, physically based workbench setting. The URP system uses pre-existing building models as input to an urban planning system (Underkoffler and Ishii 1999). MIXDesign allows architects to interact with a real scale model 6 M. J. KIM AND M. L. MAHER of the design by using a paddle in a normal working setting, and also presents an enhanced version of the scale model with 3D virtual objects registered to the real ones (Dias et al. 2002). ARTHUR system is an Augmented Round Table for architecture and urban planning, where virtual 3D objects are projected into the common working environment by semitransparent stereoscopic head mounted display (HMDs). Placeholder objects (PHOs) and wand are used to control virtual objects (Granum et al. 2003). These various configurations of tabletop systems, with and without AR, show a trend in developing technology. The different configurations described above draw on specific intended uses to define the components and their configuration. Few of the publications about digital workbenches evaluate the new interface technology with respect to spatial cognition or improved understanding of the spatial relationships of the components of the digital model. While TUIs and GUIs will continue to be alternative design environments for digital models, we focus on the differences between them in order to clarify the benefit of TUIs for designers. 3. Experiment Setting: GUI-based and TUI-based Collaboration In devising an experiment that can highlight the expected improvement in spatial cognition while using TUIs, we chose to compare design collaboration in the following settings: TUIs on a tabletop design environment and GUIs on a desktop design environment. We expect that this comparison will enable us to verify if and in what way TUIs affect designers’ spatial understanding of 3D models in computer-mediated collaborative design. 3.1. DESIGN COLLABORATION IN A GUI ENVIRONMENT The setting of the GUI design environment is a desktop computer with a GUI such as a mouse, a keyboard and a LCD screen shown in table 1. TABLE 1. GUI design environment Hardware Desktop computer/ Mouse & Keyboard Application ArchiCAD Display space Vertical LCD screen Task space Mouse & keyboard